Project 1: Images of the Russian Empire - Colorizing the Prokudin-Gorskii photo collection

Tyler Yang

Overview

Prokudin Gorskii was a pioneer of his time. He believed that color photography was going to take over the future. However, he knew that there was not way to print out the the color photos out at his time. Thus, he came up with the brilliant idea of taking three photos of the same scene using a red, green, and blue filter. An example of one of these photos is pictured below.

Figure: A picture of a russian cathedral taken by Prokudin Gorskii.

The goal of this project is to be able to take these sets of three photos and turn them into a color image using the technology we have today.

Solution

To do this, I first seperate the three channels. The top photo is the blue filter, proceeded by the green filter, and finally the red filter. In order to align the channels together, I utilize a metric called Normalized Cross-Correlation (NCC) where if I turn the images into vectors, I can simply perform a dot product between the two normalized vectors to see similiarity. The higher the dot product, the more similar the images are. We utilize a circular shift to determine the offset needed to perform the correct alignment. The process to check alignment are as follows:

(1) crop the channels to only use the middle 50% for NCC calculation. We will get undesired effects if we allow the black border to be included in our NCC metric (especially if it rolls over from our circular shift).

(2) For an exhaustive search of an offset from (-30, -30) to (30, 30), we will circular shift and calculate NCC metrics for each offset.

(3) The offset with the highest NCC metric is the offset we will use to align the channels.

It is also worth noting that I used the red filter as a reference and set offsets for blue and green.

With larger images - this single scale exhaustive search method is not efficient. Thus, I moved to implement a course-to-fine pyramid speedup method. For this, I resize the image until the maximum dimension is less than 320 pixels. Then I called the exhaustive search function mentioned above. Afterwards, I resize the image and the calculated offset - scaling upwards by a factor of 2. At each step, I call the align function again with the new resolution and refine the offset by running an exhaustive search from the previous offset with a margin of 2.

This method was enough to get sufficient alignment results on all images but one which I will talk about in the next section. Below I will showcase all images in the dataset with an offset table below:

Cathedral: B Offset: (-7, -1), G Offset: (-12, -3)

Church: B Offset: (-33, 8), G Offset: (-58, 4)

Emir: B Offset: (-57, -17), G Offset: (40, -13)

Harvesters: B Offset: (-64, 3), G Offset: (-124, -14)

Icon: B Offset: (-49, -5), G Offset: (-90, -23)

Lady: B Offset: (-60, -4), G Offset: (-112, -12)

Melons: B Offset: (-96, -3), G Offset: (-180, -13)

Monastery: B Offset: (-6, -1), G Offset: (-3, -2)

Onion Church: B Offset: (-57, -10), G Offset: (-108, -36)

Sculpture: B Offset: (-107, 16), G Offset: (-140, 27)

Self Portrait: B Offset: (-98, -8), G Offset: (-176, -37)

Three Generations: B Offset: (-59, 3), G Offset: (-111, -12)

Tobolsk: B Offset: (-4, -1), G Offset: (-7, -3)

Train: B Offset: (-43, -26), G Offset: (-87, -32)

Bells and Whistles

Notice that there IS indeed a photo that in the last section was not 100 percent aligned. No matter how much I modified parameters, I could not seem to get emir.tif to be aligned. The main problem with emir.tif lied in the fact that Blue was just drastically more emphasized than Red or Green causing NCC to not work as well. I decided to try a drastically different metric than NCC - SSIM (The Structural Similarity Index). Rather than using RGB, fromWang et al.,2004SSIM utilizes luminance, constrast, and structure to determine similarity. An approximation is made within the paper such that we can utilize the formula: SSIM Formula where x and y are the two channels, mu is the sample mean, and C_1 = (k_1L)^2, C_2 = (k_2L)^2, where k_1 = 0.01, k_2 = 0.03, and L = 255. However, after implementing SSIM, I was still slightly dissapointed with the results, so I tried aligning with edges.

I utilized the sobel edge detection to find edges in the image. The Sobel operator takes advantage of the fact that numerically - in a gray scale image, edges are represented by fast changes in pixel intensity. Hence, we can use gradients to actually determine where edges are! At edges gradients will be larger, while at other locations, gradients will be smaller. A great explanation of this and visualization can also be found through theOpenCV documentation.

Thus, I turned all blue, green and red channels into sobel edge detected images. Then, I simply ran the same course-to-fine image pyramid alignment method as before on these new images. The results are astounding.

Before Emir: B Offset: (-57, -17), G Offset: (40, -13)